Papers by Matija Luka Kukić
Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Modern language models (LMs) are trained in autoregressive manner, conditioned on the prefix. sequence labeling (SL) tasks assign labels to each individual input token, naturally benefiting from bidirectional context. |
| Approach: | They explore sequence repetition (SR) as a less invasive alternative to decoder-only models . they show that increasing the number of repetitions does not degrade SL performance . |
| Outcome: | The proposed technique improves the quality of token-level embeddings and surpasses encoders and unmasked decoders. |